Heterogeneity and statistical significance in meta-analysis: an empirical study of 125 meta-analyses

2000 ◽  
Vol 19 (13) ◽  
pp. 1707-1728 ◽  
Author(s):  
Eric A. Engels ◽  
Christopher H. Schmid ◽  
Norma Terrin ◽  
Ingram Olkin ◽  
Joseph Lau
Keyword(s):  
Empirical Study ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Meta Analyses

scholarly journals Incidence of Respiratory Symptoms for Residents Living Near a Petrochemical Industrial Complex: A Meta-Analysis

2020 ◽  
Vol 17 (7) ◽  
pp. 2474
Author(s):  
Wen-Wen Chang ◽  
Hathaichon Boonhat ◽  
Ro-Ting Lin
Keyword(s):  
Respiratory Symptoms ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Random Effect ◽  
Study Groups ◽  
Industrial Complex ◽  
Residential Exposure ◽  
Increased Risk ◽  
Subgroup Analyses ◽  
Meta Analyses

The air pollution emitted by petrochemical industrial complexes (PICs) may affect the respiratory health of surrounding residents. Previous meta-analyses have indicated a higher risk of lung cancer mortality and incidence among residents near a PIC. Therefore, in this study, a meta-analysis was conducted to estimate the degree to which PIC exposure increases the risk of the development of nonmalignant respiratory symptoms among residents. We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines to systematically identify, select, and critically appraise relevant research. Finally, we identified 16 study groups reporting 5 types of respiratory symptoms: asthma, bronchitis, cough, rhinitis, and wheezing. We estimated pooled odds ratios (ORs) using random-effect models and investigated the robustness of pooled estimates in subgroup analyses by location, observation period, and age group. We determined that residential exposure to a PIC was significantly associated with a higher incidence of cough (OR = 1.35), wheezing (OR = 1.28), bronchitis (OR = 1.26), rhinitis (OR = 1.17), and asthma (OR = 1.15), although the latter two associations did not reach statistical significance. Subgroup analyses suggested that the association remained robust across different groups for cough and bronchitis. We identified high heterogeneity for asthma, rhinitis, and wheezing, which could be due to higher ORs in South America. Our meta-analysis indicates that residential exposure to a PIC is associated with an increased risk of nonmalignant respiratory symptoms.

Can Reliance be Placed on a Single Meta-Analysis?

1990 ◽  
Vol 24 (3) ◽  
pp. 405-415 ◽  
Author(s):  
Nathaniel McConaghy
Keyword(s):  
Literature Review ◽  
Effect Size ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Effect Sizes ◽  
Control Groups ◽  
Consistent Finding ◽  
Placebo Controls ◽  
Effect Of Treatment ◽  
Meta Analyses

Meta-analysis replaced statistical significance with effect size in the hope of resolving controversy concerning evaluation of treatment effects. Statistical significance measured reliability of the effect of treatment, not its efficacy. It was strongly influenced by the number of subjects investigated. Effect size as assessed originally, eliminated this influence but by standardizing the size of the treatment effect could distort it. Meta-analyses which combine the results of studies which employ different subject types, outcome measures, treatment aims, no-treatment rather than placebo controls or therapists with varying experience can be misleading. To ensure discussion of these variables meta-analyses should be used as an aid rather than a substitute for literature review. While meta-analyses produce contradictory findings, it seems unwise to rely on the conclusions of an individual analysis. Their consistent finding that placebo treatments obtain markedly higher effect sizes than no treatment hopefully will render the use of untreated control groups obsolete.

scholarly journals Spinal Pedicle Screw Insertion

2018 ◽  
Vol 27 (4) ◽  
pp. 307-312
Author(s):  
Rafaella Do Rosario Tacla ◽  
Murilo Henrique Guedes ◽  
Lize Mattiello Oliveira ◽  
Luiz Roberto Aguiar
Keyword(s):  
Pedicle Screw ◽  
Odds Ratio ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Screw Insertion ◽  
Guided Surgery ◽  
Pubmed Database ◽  
Spinal Region ◽  
Meta Analyses ◽  
Navigation Method

We aimed to verify the reproducibility of the results obtained from previous systematic reviews and meta-analyses, focusing specifically on methodological and technical aspects, such as navigation method, screw insertion technique and spinal region involved to provide a more homogenous analysis. The PubMed database was searched using three terms: “stereotaxic techniques”, “neuronavigation”, and “image-guided surgery,” each associated with other two terms, “pedicle” and “screw”. Data were transferred to Comprehensive Meta-Analysis software, v.3, for statistical analysis. The significance of the pooled odds ratio was determined by the Z test, along with95% confidence intervals. A funnel plot was used to assess publication bias. The statistical significance was defined as p ≤ 0.05. The meta-analysis yielded an overall odds ratio of 2.9 favouring navigation. The average correct positioning of the screws was 91.8% for navigation, while 82.2% of screws were deemed appropriate in fluoroscopy. The odds ratio obtained for the thoracic and lumbar spines separately was 3.1 and 2.7, respectively. Among the fluoroscopy guided surgeries, 53 complications occurred, whereas only eleven complications were observed among those guided by neuronavigation. This meta-analysis suggests that compared with conventionalfluoroscopy technique, neuronavigation allows for more accurate pedicle screw placement.

scholarly journals Influence of Pilot and Small Trials in Meta-Analyses of Behavioral Interventions: A Meta-epidemiological Study

2020 ◽  
Author(s):  
Michael W. Beets ◽  
R. Glenn Weaver ◽  
John P.A. Ioannidis ◽  
Alexis Jones ◽  
Lauren von Klinggraeff ◽  
...  
Keyword(s):  
Sample Size ◽  
Behavioral Interventions ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Feasibility Studies ◽  
Small Sample ◽  
Effect Sizes ◽  
Absolute Difference ◽  
Meta Analyses ◽  
The Impact

Abstract Background: Pilot/feasibility or studies with small sample sizes may be associated with inflated effects. This study explores the vibration of effect sizes (VoE) in meta-analyses when considering different inclusion criteria based upon sample size or pilot/feasibility status. Methods: Searches were conducted for meta-analyses of behavioral interventions on topics related to the prevention/treatment of childhood obesity from 01-2016 to 10-2019. The computed summary effect sizes (ES) were extracted from each meta-analysis. Individual studies included in the meta-analyses were classified into one of the following four categories: self-identified pilot/feasibility studies or based upon sample size (N≤100, N>100, and N>370 the upper 75th of sample size). The VoE was defined as the absolute difference (ABS) between the re-estimations of summary ES restricted to study classifications compared to the originally reported summary ES. Concordance (kappa) of statistical significance between summary ES was assessed. Fixed and random effects models and meta-regressions were estimated. Three case studies are presented to illustrate the impact of including pilot/feasibility and N≤100 studies on the estimated summary ES.Results: A total of 1,602 effect sizes, representing 145 reported summary ES, were extracted from 48 meta-analyses containing 603 unique studies (avg. 22 avg. meta-analysis, range 2-108) and included 227,217 participants. Pilot/feasibility and N≤100 studies comprised 22% (0-58%) and 21% (0-83%) of studies. Meta-regression indicated the ABS between the re-estimated and original summary ES where summary ES were comprised of ≥40% of N≤100 studies was 0.29. The ABS ES was 0.46 when summary ES comprised of >80% of both pilot/feasibility and N≤100 studies. Where ≤40% of the studies comprising a summary ES had N>370, the ABS ES ranged from 0.20-0.30. Concordance was low when removing both pilot/feasibility and N≤100 studies (kappa=0.53) and restricting analyses only to the largest studies (N>370, kappa=0.35), with 20% and 26% of the originally reported statistically significant ES rendered non-significant. Reanalysis of the three case study meta-analyses resulted in the re-estimated ES rendered either non-significant or half of the originally reported ES. Conclusions: When meta-analyses of behavioral interventions include a substantial proportion of both pilot/feasibility and N≤100 studies, summary ES can be affected markedly and should be interpreted with caution.

scholarly journals Women’s decision in family planning use and its determinants in Ethiopia. A systematic review and meta-analysis protocol

2021 ◽  
Author(s):  
Etsay Woldu Anbesu ◽  
Setognal Birara Aychiluhm ◽  
Mussie Alemayehu
Keyword(s):  
Systematic Review ◽  
Decision Making ◽  
Family Planning ◽  
Contraceptive Use ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Grey Literature ◽  
Reproductive Health Services ◽  
Mesh Terms ◽  
Meta Analyses

Abstract Background Low uptake of contraceptive use have much consequence on maternal and child health. Despite this effect, less emphasis was given to women’s decision-making on family planning use in Ethiopia. Though there are studies conducted in different parts of the country on women’s decision-making in family planning use, there are inconsistent findings. Thus, this systematic review and meta-analysis aimed to determine the pooled prevalence of women’s decision-making in family planning use and its determinants in Ethiopia. Methods Preferred Reporting Items for Systematic Reviews and Meta-Analyses guideline will be followed to develop the review protocol. All observational studies will be retrieved using Medical Subject Heading (Mesh) terms or keywords from online databases PubMed, CINAHL, Google Scholar, and unpublished grey literature from May 10-11August/2021. The quality of the studies will be critically assessed using Joanna Briggs Institute checklist. Heterogeneity among studies will be examined using I-squared statistics. Funnel plots and Egger’s test will be used to examine publication bias. The Meta-analysis will be performed using STATA version 14 software. Statistical significance will be determined at 95% CI. Discussion Improving women’s autonomy on decision-making on reproductive health services including contraceptive use has a substantial advantage. There are studies on women’s decision-making in family planning use and this systematic review and meta-analysis will help policymakers to develop appropriate interventions.

scholarly journals Migraine and Medical Ramifications: A Comprehensive Overview Based on Observational Study Meta-Analyses

2021 ◽  
Vol 12 ◽  
Author(s):  
Weiwei Chen ◽  
Wenqi Qian ◽  
Lixian Zhong ◽  
Gongwei Jing
Keyword(s):  
Meta Analysis ◽  
Statistical Significance ◽  
Umbrella Review ◽  
Comprehensive Overview ◽  
End Point ◽  
Summary Effect ◽  
Evidence Quality ◽  
Evidence Grade ◽  
Meta Analyses

Purpose: An umbrella review was conducted for comprehensively evaluating previous review-based literature together with meta-analysis of observational investigations probing correlations between migraine and medical end-point ramifications in patients. The breadth and validity of these associations were assessed.Methods: Multiple online scientific repositories (including PubMed, Medline, Embase, and Web of Science) were investigated (inception-August 2021) for related meta-analyses focusing on links between migraine and all possible health/medical ramification end-points. A summary effect size and 95% CIs were determined for each identified study with such links. Heterogeneity and small-study influence traces were also evaluated. The AMSTAR 2 platform was employed for evaluating standards of methodology, together with objective criteria, for assessing the standards of datasets from each medical end-point scrutinized in this study.Results: A total of 25 scientific reports comprising 10,237,230 participants for 49 meta-analyses of observational studies were selected. Among such 49 outcomes, 30 demonstrated statistical significance (P < 0.05). Significant associations were observed in multiple diseases, including cardiovascular/cerebrovascular, cerebral, pregnancy-related and metabolic disorders, other outcomes, and mortality.Conclusion: The results showed that migraine increased the risk of 29 health outcomes, though lowered the risk of breast cancer. However, evidence quality was graded as high only for angina. The evidence quality of ischaemic stroke, stroke, MACCE, WMAs, and asthma was graded as moderate. All remaining 24 outcomes had an evidence grade of “weak.”

scholarly journals Low statistical power in biomedical science: a review of three human research domains

2017 ◽  
Vol 4 (2) ◽  
pp. 160254 ◽  
Author(s):  
Estelle Dumas-Mallet ◽  
Katherine S. Button ◽  
Thomas Boraud ◽  
Francois Gonon ◽  
Marcus R. Munafò
Keyword(s):  
Effect Size ◽  
Statistical Power ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Average Power ◽  
Biomedical Science ◽  
Significant Finding ◽  
Biomedical Sciences ◽  
True Effect Size ◽  
Meta Analyses

Studies with low statistical power increase the likelihood that a statistically significant finding represents a false positive result. We conducted a review of meta-analyses of studies investigating the association of biological, environmental or cognitive parameters with neurological, psychiatric and somatic diseases, excluding treatment studies, in order to estimate the average statistical power across these domains. Taking the effect size indicated by a meta-analysis as the best estimate of the likely true effect size, and assuming a threshold for declaring statistical significance of 5%, we found that approximately 50% of studies have statistical power in the 0–10% or 11–20% range, well below the minimum of 80% that is often considered conventional. Studies with low statistical power appear to be common in the biomedical sciences, at least in the specific subject areas captured by our search strategy. However, we also observe evidence that this depends in part on research methodology, with candidate gene studies showing very low average power and studies using cognitive/behavioural measures showing high average power. This warrants further investigation.

scholarly journals An Updated Systematic Review and Meta-Analysis of the Obstetric Consequences of Female Genital Mutilation/Cutting

2014 ◽  
Vol 2014 ◽  
pp. 1-8 ◽  
Author(s):  
R. C. Berg ◽  
J. Odgaard-Jensen ◽  
A. Fretheim ◽  
V. Underland ◽  
G. Vist
Keyword(s):  
Systematic Review ◽  
Prospective Studies ◽  
Retrospective Studies ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Previous Systematic Review ◽  
Genital Mutilation ◽  
Meta Analyses ◽  
Female Genital ◽  
Made In

In our recent systematic review inObstetrics and Gynecology Internationalof the association between FGM/C and obstetric harm we concluded that FGM/C significantly increases the risk of delivery complications. The findings were based on unadjusted effect estimates from both prospective and retrospective studies. To accommodate requests by critics, we aimed to validate these results through additional analyses based on adjusted estimates from prospective studies. We judged that 7 of the 28 studies included in our original systematic review were prospective. Statistical adjustments for measured confounding factors were made in eight studies, including three prospective studies. The adjusted confounders differed across studies in number and type. Results from meta-analyses based on adjusted estimates, with or without data from retrospective studies, consistently pointed in the same direction as our earlier findings. There were only small differences in the sizes or the level of statistical significance. Using GRADE, we assessed that our confidence in the effect estimates was very low or low for all outcomes. The adjusted estimates generally show similar obstetric harms from FGM/C as unadjusted estimates do. Thus, the current analyses confirm the findings from our previous systematic review. There are sufficient grounds to conclude that FGM/C, with respect to obstetric circumstances, involves harm.

scholarly journals Effects of Discontinuation of Drugs Used for Augmentation Therapy on Treatment Outcomes in Depression: A Systematic Review and Meta-analysis

2020 ◽  
Author(s):  
Hideo Kato ◽  
Teruki Koizumi ◽  
Hiroyoshi Takeuchi ◽  
Hideaki Tani ◽  
Masaru Mimura ◽  
...  
Keyword(s):  
Treatment Outcomes ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Augmentation Therapy ◽  
Double Blind ◽  
Randomized Controlled ◽  
Treatment Of Depression ◽  
Meta Analyses ◽  
Stabilization Period ◽  
Frequent Relapse

Abstract Introduction There has been no consensus on whether and how long add-on drugs for augmentation therapy should be continued in the treatment of depression. Methods Double-blind randomized controlled trials that examined the effects of discontinuation of drugs used for augmentation on treatment outcomes in patients with depression were identified. Meta-analyses were performed to compare rates of study withdrawal due to any reason, study-defined relapse, and adverse events between patients who continued augmentation therapy and those who discontinued it. Results Seven studies were included (n=841 for continuing augmentation therapy; n=831 for discontinuing augmentation therapy). The rate of study withdrawal due to any reason was not significantly different between the 2 groups (risk ratio [RR]=0.86, 95% confidence interval [CI]=0.69–1.08, p=0.20). Study withdrawal due to relapse was less frequent in the continuation group than in the discontinuation group (RR=0.61, 95% CI=0.40–0.92, p=0.02); however, this statistical significance disappeared when one study using esketamine as augmentation was excluded. Analysis of the data from 5 studies that included a stabilization period before randomization found less frequent relapse in the continuation group than in the discontinuation group (RR=0.47, 95% CI=0.36–0.60, p<0.01). This finding was repeated when the esketamine study was excluded. Discussion No firm conclusions could be drawn in light of the small number of studies included. Currently available evidence suggests that add-on drugs, other than esketamine, used for augmentation therapy for depression may be discontinued. This may not be the case for patients who are maintained with augmentation therapy after remission.

scholarly journals Cannons and sparrows II: the enhanced Bernoulli exact method for determining statistical significance and effect size in the meta-analysis of k 2 × 2 tables

2021 ◽  
Vol 18 (1) ◽  
Author(s):  
Lawrence M. Paul
Keyword(s):  
Effect Size ◽  
Type I Error ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Statistical Error ◽  
Exact Method ◽  
Type I ◽  
Exact Test ◽  
Inverse Variance ◽  
Meta Analyses

Abstract Background The use of meta-analysis to aggregate the results of multiple studies has increased dramatically over the last 40 years. For homogeneous meta-analysis, the Mantel–Haenszel technique has typically been utilized. In such meta-analyses, the effect size across the contributing studies of the meta-analysis differs only by statistical error. If homogeneity cannot be assumed or established, the most popular technique developed to date is the inverse-variance DerSimonian and Laird (DL) technique (DerSimonian and Laird, in Control Clin Trials 7(3):177–88, 1986). However, both of these techniques are based on large sample, asymptotic assumptions. At best, they are approximations especially when the number of cases observed in any cell of the corresponding contingency tables is small. Results This research develops an exact, non-parametric test for evaluating statistical significance and a related method for estimating effect size in the meta-analysis of k 2 × 2 tables for any level of heterogeneity as an alternative to the asymptotic techniques. Monte Carlo simulations show that even for large values of heterogeneity, the Enhanced Bernoulli Technique (EBT) is far superior at maintaining the pre-specified level of Type I Error than the DL technique. A fully tested implementation in the R statistical language is freely available from the author. In addition, a second related exact test for estimating the Effect Size was developed and is also freely available. Conclusions This research has developed two exact tests for the meta-analysis of dichotomous, categorical data. The EBT technique was strongly superior to the DL technique in maintaining a pre-specified level of Type I Error even at extremely high levels of heterogeneity. As shown, the DL technique demonstrated many large violations of this level. Given the various biases towards finding statistical significance prevalent in epidemiology today, a strong focus on maintaining a pre-specified level of Type I Error would seem critical. In addition, a related exact method for estimating the Effect Size was developed.

Sign in / Sign up

Export Citation Format

Share Document